Information processing apparatus, information processing system, and information processing method

ABSTRACT

In an information processing apparatus, a posture data acquiring section acquires output values from a motion sensor of a head-mounted display. Effective rates of acceleration are then obtained by correcting the output values based on correction formulas. A posture is calculated in reference to the axes of the motion sensor based on the acquired rates of acceleration. Thereafter, the posture of the head-mounted display is calculated by performing rotation correction based on previously acquired angle errors, the calculated posture being output to an information processing section. A correction data updating section acquires posture information based on markers of the head-mounted display imaged in a captured image, and updates correction data for use in acquiring the posture from the sensor output values through comparison with the marker-based posture information.

TECHNICAL FIELD

The present invention relates to information processing technology forprocessing information in response to movements in the real world.

BACKGROUND ART

There exist video games that involve capturing by a camera an image of auser's body and markers attached thereto and replacing a display regioncorresponding to the captured image with another image for display on adisplay device (e.g., see PTL 1). The techniques for analyzing not onlythe captured image but also measurements from various sensors attachedto or held by the user and having the results of the analysis reflectedin information processing such as games have gained widespreadacceptance in extensive fields ranging from small game machines toleisure facilities.

CITATION LIST Patent Literature

[PTL 1] European Published Patent No. EP0999518

SUMMARY Technical Problems

For the above-mentioned techniques, it is always a major challenge howto accurately acquire information related to the posture and movementsof a target object. For example, where a head-mounted display worn by auser on the head presents the user with a virtual world reflecting notonly movements of the head but also changes in the user's visual field,how accurately to detect the posture and movements of the head is achallenge to be addressed in bringing about the expressions of a morerealistic sensation and a higher sense of immersion. In addition to theproblem of the expressions, detected values even slightly inconsistentwith the posture of the head can cause the user to lose the sense ofequilibrium during continuous viewing of the display image reflectingthe detected values. This can be dangerous enough to adversely affectthe user's physical condition.

The present invention has been made in view of the above circumstances.An object of the invention is therefore to provide techniques foracquiring information related to the posture of a target object easilyand highly accurately.

Solution to Problems

According to one embodiment of the present invention, there is providedan information processing apparatus. The information processingapparatus includes an input information acquiring section configured toacquire output values from an acceleration sensor mounted on a targetobject, a posture data acquiring section configured to acquire postureinformation related to the target object by calculating rates ofacceleration on axes of the acceleration sensor based on the outputvalues so as to obtain inclination angles of the axes based on thecalculated rates of acceleration, and an information processing sectionconfigured to perform information processing using the postureinformation before outputting output data generated as a result of theinformation processing. On the basis of angle errors between the axes ofthe acceleration sensor on the one hand and reference axes given to thetarget object on the other hand, the posture data acquiring sectioncorrects the posture information related to the acceleration sensor inaccordance with the posture information related to the target object.

According to another aspect of the present invention, there is providedan information processing system. The information processing systemincludes a head-mounted display, and an information processing apparatusconfigured to establish connection with the head-mounted display and togenerate data of a display image. The information processing apparatusincludes an input information acquiring section configured to acquireoutput values from an acceleration sensor mounted on the head-mounteddisplay, a posture data acquiring section configured to acquire postureinformation related to the head-mounted display by calculating rates ofacceleration on axes of the acceleration sensor based on the outputvalues so as to obtain inclination angles of the axes based on thecalculated rates of acceleration, and an information processing sectionconfigured to generate the data of the display image having a visualview being changed in accordance with posture using the postureinformation, the information processing section further outputting thedisplay image data to the head-mounted display. The posture dataacquiring section corrects the posture information related to theacceleration sensor in accordance with the posture information relatedto the head-mounted display on the basis of angle errors between theaxes of the acceleration sensor on the one hand and reference axes givento the head-mounted display on the other hand.

According to a further aspect of the present invention, there isprovided an information processing method. The information processingmethod includes the steps of acquiring output values from anacceleration sensor mounted on a target object, acquiring postureinformation related to the target object by calculating rates ofacceleration on axes of the acceleration sensor based on the outputvalues so as to obtain inclination angles of the axes based on thecalculated rates of acceleration, and performing information processingusing the posture information before outputting output data generated asa result of the information processing to another apparatus. On thebasis of angle errors between the axes of the acceleration sensor on theone hand and reference axes given to the target object on the otherhand, the posture information acquiring step corrects the postureinformation related to the acceleration sensor in accordance with theposture information related to the target object.

Incidentally, if other combinations of the above-outlined composingelements or the above expressions of the present invention are convertedbetween different forms such as a method, an apparatus, a system, acomputer program, and a recording medium that records the computerprogram, they still constitute effective embodiments of the presentinvention.

Advantageous Effect of Invention

According to the present invention, the information related to theposture of the target object is obtained easily and highly accurately.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram depicting a typical configuration of aninformation processing system to which an embodiment of the presentinvention may be applied.

FIG. 2 is a block diagram depicting a typical internal circuitconfiguration of an information processing apparatus as one embodimentof the present invention.

FIG. 3 is a block diagram depicting a typical internal circuitconfiguration of a head-mounted display (HMD) in the embodiment of thepresent invention.

FIG. 4 is a schematic diagram depicting an external shape of the HMD inthe embodiment of the present invention.

FIG. 5 is a block diagram depicting a configuration of functional blocksin the information processing apparatus as one embodiment of the presentinvention and in the HMD constituting a part thereof.

FIG. 6 is a schematic diagram explaining typical relations betweendisplay images on the one hand and rates of acceleration targeted formeasurement with an acceleration sensor in the embodiment on the otherhand.

FIG. 7 is a schematic diagram explaining the technique for obtainingcorrection formulas through calibration with the embodiment.

FIG. 8 is a schematic diagram explaining the principle of performingrotation correction with the embodiment.

FIG. 9 is a flowchart depicting the procedure of a posture acquiringprocess performed by a posture data acquiring section of the embodiment.

DESCRIPTION OF EMBODIMENT

FIG. 1 depicts a typical configuration of an information processingsystem to which an embodiment of the present invention may be applied.An information processing system 8 includes an imaging device 12 thatimages the target object, an information processing apparatus 10 thatperforms information processing based on captured images, a flat-screendisplay device 16 that displays images resulting from the informationprocessing, a head-mounted display (called the HMD hereunder) 18 thatalso displays such images, and an input device 14 operated by a user.

The information processing apparatus 10, the imaging device 12, theinput device 14, the flat-screen display device 16, and the HMD 18 maybe interconnected by cable or by known wireless communication technologysuch as Bluetooth (registered trademark). Depending on the informationprocessing performed by the information processing apparatus 10, theinput device 14 and the flat-screen display device 16 may be selectivelyintroduced. The external shapes of these devices are not limited to whatis illustrated in the drawing. At least two of the above-mentioneddevices may be integrally formed. For example, the informationprocessing apparatus 10, the input device 14, and the flat-screendisplay device 16 may be formed integrally in a portable terminal.

The HMD 18 is a display device worn by the user on the head and designedto display images on a display panel such as an organicelectroluminescent (EL) panel positioned in front of the user's eyes.For example, the HMD 18 may present a stereoscopic image by generatingparallax images as viewed from right and left points of view and bydisplaying the generated parallax images on bisected right and leftregions of a display screen. Motion sensors such as an accelerationsensor and a gyro sensor are incorporated in or externally attached tothe HMD 18. These sensors measure physical quantities related to theposture and movements of the user's head and transmit the measurementsconsecutively to the information processing apparatus 10. The HMD 18 mayfurther incorporate speakers or earphones positioned against the user'sears for audio output thereto.

The imaging device 12 includes a camera that images the target objectsuch as the user at a predetermined frame rate and a mechanism thatgenerates output data of a captured image by performing known processessuch as demosaicing on the signal output from the camera, the mechanismfurther outputting the generated output data to the informationprocessing apparatus 10. The camera is a stereo camera that includescommon visible light sensors such as charge-coupled device (CCD) sensorsor complementary metal oxide semiconductor (CMOS) sensors positionedright and left at a predetermined distance apart.

The information processing apparatus 10 generates output data such asimages and sounds by performing necessary information processing usingmeasurements sent from the motion sensors of the HMD 18 and capturedimage data from the imaging device 12. The content of the processingperformed by the information processing apparatus 10 is not limited toanything specific and may be determined as needed depending on thedetailed functions and applications desired by the user. For example, avideo game may be advanced in accordance with the movements and theinput made by the user. Display images of the virtual world may begenerated in such a manner that the visual field of the virtual worldwill change in keeping with head movements. The user's movements may beconverted to command input for information processing.

The information processing apparatus 10 may acquire movements of theinput device 14 using markers attached thereto. The informationprocessing apparatus 10 may alternatively acquire the posture andmovements of the user's head using means other than the motion sensors,such as by tracking multiple markers mounted on an external surface ofthe HMD 18. In such a case, the measurements from the motion sensors andthose from the other means may be utilized in a complementary manner toimprove the accuracy in detecting the posture and movements. As anotheralternative, as will be discussed later, information related to thehead's posture acquired from marker images in a captured image may beused to correct the measurements from the motion sensors. The outputdata generated by the information processing apparatus 10 is transmittedat least either to the HMD 18 or to the flat-screen display device 16.

The flat-screen display device 16 may be a television (TV) set that hasa display for outputting two-dimensional images and speakers foroutputting sounds. For example, the flat-screen display device 16 may bea liquid crystal display TV set, an organic EL TV set, a plasma displayTV set, or a personal computer (PC) display. In another example, theflat-screen display device 16 may be the display of a tablet terminal ora mobile terminal with speakers. The input device 14, when operated bythe user, receives requests such as those for starting and endingprocesses, selecting functions, and inputting commands, and outputs thereceived request to the information processing apparatus 10 as anelectrical signal.

The input device 14 may be any one of common input devices such as agame controller, a keyboard, a mouse, a joystick, a touch pad mounted onthe display surface of the flat-screen display device 16, or acombination of these devices. The input device 14 may further include alight-emitting marker having an element emitting light in apredetermined color, or an aggregate of such light-emitting elements. Inthis case, the information processing apparatus 10 may track themovement of the marker using captured images and interpret the movementof the input device 14 as the user's operation. As another alternative,the input device 14 may be composed of only a light-emitting marker anda mechanism for holding that marker.

FIG. 2 depicts an internal circuit structure of the informationprocessing apparatus 10. The information processing apparatus 10includes a central processing unit (CPU) 22, a graphics processing unit(GPU) 24, and a main memory 26. These components are interconnected viaa bus 30. The bus 30 is also connected to an input/output interface 28.The input/output interface 28 is connected to peripheral deviceinterfaces, such as a universal serial bus (USB) interface and anInstitute of Electrical and Electronics Engineers (IEEE) 1394 interface;a communication section 32 made of a wired or wireless local areanetwork (LAN) interface; a storage section 34, such as a hard disk driveor a nonvolatile memory; an output section 36 for outputting data to theflat-screen display device 16 and the HMD 18; an input section 38 forinputting data from the imaging device 12, the input device 14, or theHMD 18; and a recording medium driving section 40 that drives removablerecording media, such as magnetic disks, optical disks, or semiconductormemories.

The CPU 22 controls the entire information processing apparatus 10 byexecuting the operating system stored in the storage section 34. The CPU22 also executes various programs read from the removable recordingmedium and loaded into the main memory 26 or programs downloaded via thecommunication section 32. The GPU 24 has the function of a geometryengine and that of a rendering processor. In accordance with a renderinginstruction from the CPU 22, the GPU 24 performs a rendering process andstores the resulting display image in a frame buffer (not depicted). TheGPU 24 proceeds to convert the display image in the frame buffer into avideo signal and output the video signal to the output section 36. Themain memory 26 is composed of a random access memory (RAM) that storesthe programs and data necessary for the processing.

FIG. 3 depicts an internal circuit structure of the HMD 18. The HMD 18includes a CPU 50, a main memory 52, a display section 54, and an audiooutput section 56. These components are interconnected via a bus 58. Thebus 58 is further connected to an input/output interface 60. Theinput/output interface 60 is connected to a communication section 62made of a wired or wireless LAN network interface, a motion sensor 64,and a light-emitting section 66.

The CPU 50 processes the information acquired from the components of theHMD 18 via the bus 58 and feeds output data to the display section 54and the audio output section 56. The main memory 52 stores the programsand data necessary for processing by the CPU 50. However, depending onthe application to be executed or the design of equipment in use, theinformation processing apparatus 10 may carry out most of theprocessing, so that the HMD 18 only needs to output the data sent fromthe information processing apparatus 10. In this case, the CPU 50 andthe main memory 52 may be replaced with simpler devices.

The display section 54 is configured with a display panel such as aliquid crystal display panel or an organic EL panel that displays imagesin front of the eyes of the user wearing the HMD 18. As mentioned above,a stereoscopic view may be implemented by displaying a pair of parallaximages in the panel regions corresponding to the right and left eyes.The display section 54 may further include a pair of lenses positionedbetween the display panel and the eyes of the user wearing the HMD 18,the lenses acting to expand a viewing angle of the user. In this case,the display image is subjected to distortion correction such that theimage will appear normally when viewed via the lenses on the informationprocessing apparatus 10 or on the HMD 18.

The audio output section 56 includes speakers or earphones positionedwhere the use's ears are located when the HMD 18 is worn by the user,allowing the user to hear sounds. The number of audio channels foroutput is not limited. The audio output section 56 may have monaural,stereo, or surround speakers or headphones. The communication section 62is an interface that transmits and receives data to and from theinformation processing apparatus 10 and the flat-screen display device16. For example, the communication section 62 may be implemented usingknown wireless communication technology such as Bluetooth (registeredtrademark).

The motion sensors 64 are provided through the combination of anacceleration sensor and a gyro sensor, for example, and detect theposture and movements of the HMD 18. The results of the detection aretransmitted to the information processing apparatus 10 via thecommunication section 62. The light-emitting section 66 is an element oran aggregate of elements emitting light in a predetermined color.Multiple light-emitting elements are attached to the external surface ofthe HMD 18. These light-emitting elements tracked as markers permitacquisition of the position of the HMD 18. Also, the posture of the HMD18 is acquired on the basis of the number of marker images and thepositional relations therebetween in the captured image.

FIG. 4 depicts the appearance of the HMD 18. In this example, the HMD 18is made up of an output mechanism section 102 and a wearing mechanismsection 104. The wearing mechanism section 104 includes a wear band 106worn by the user around the head to secure the device. The wear band 106is made of a material adjustable in length to the circumference of theuser's head or has such a structure. For example, the wear band 106 maybe formed by an elastic body such as rubber or may employ a buckle orgear arrangement.

The output mechanism section 102 includes an enclosure 108 shaped tocover the user's right and left eyes when the HMD 18 is worn by theuser. Inside the enclosure 108 is a display panel facing both eyes whenthe device is worn. Outside the enclosure 108 are light-emitting markers110 a, 110 b, 110 c, and 110 d. Although the number of light-emittingmarkers and their locations are not limited, four light-emitting markersare arranged at the four corners of the enclosure front of the outputmechanism section 102 in the present embodiment. Further, light-emittingmarkers 110 e and 110 f may also be arranged on both sides at the backof the wear band 106. The light-emitting markers 110 c and 110 d underthe output mechanism section 102 and the light-emitting markers 110 eand 110 f outside the wear band 106 are not seen from the view of FIG. 4and are thus indicated with broken lines depicting the circumferences ofeach marker.

The light-emitting markers indicate the position of the user on a planeof the captured image. When mounted on the HMD 18 as illustrated, thelight-emitting markers are also used to acquire the posture of the head.For example, if the user faces straight at the imaging device 12, fourlight-emitting markers 110 a, 110 b, 110 c, and 110 d are imaged. If theuser faces sideways, three light-emitting markers (e.g., light-emittingmarkers 110 b, 110 d, and 110 e) are imaged. If the user facesbackwards, then two light-emitting markers 110 e and 110 f are imaged.

Even if the user faces somewhere between these directions, the postureof the head in a three-dimensional (3-D) space of the HMD 18 can beestimated on the basis of the positional relations between marker imagesin the captured image for example, provided that the relationshipbetween the posture of a 3-D model of the HMD 18 on the one hand andprojections of the markers on an imaging plane on the other hand isobtained in the same manner as in computer graphics. The motion sensors64, not depicted, are further incorporated in or externally attached tothe HMD 18. What is fundamental for the present embodiment is to obtainboth the angle relative to a gravity vector using the accelerationsensor as part of the motion sensors 64 and the direction of the faceusing the gyro sensor as another part of the motion sensors 64, andacquire the posture of the user's head.

The position of the head in the 3-D space is acquired on the basis ofthe positions of the marker images in the stereoscopic image captured bythe stereo camera from the right and left points of view. When theposition and posture of the head are acquired in this manner and thevisual field of a virtual-world image displayed on the HMD 18 is variedaccordingly, the user can experience the sensation of almost beinginside the virtual world. When the posture is acquired, the accuracy ofthe acquisition may be improved by integrally obtaining the informationrelated to the marker images in the captured image. The HMD 18 and themotion sensors 64 may not be integrally formed. For example, the motionsensors 64 may be attached to parts of the user's body so that theposture and movements of the body are detected thereby and turned intoan image. The image thus generated may be displayed on the flat-screendisplay device 16. This also constitutes a setup to which the presentembodiment may be applied.

In any setup, the measurements from the motion sensors 64, when evenslightly inconsistent with reality, can significantly affect informationprocessing and the resulting display. For example, in a displayedvirtual world of a video game, the user may intend to advance straightbut may find himself or herself moving in a slightly differentdirection, or the user may keep the head straight up but the visualfield of the display image may appear slightly inclined. This can givethe user a feeling of discomfort. Continuously watching a horizontallyunbalanced image can cause the user to lose the sense of equilibrium andcan even adversely affect the user's physical condition. In the presentembodiment, correction is performed appropriately over an elapse of timebetween output events of values from one included sensor to another suchas the acceleration sensor and gyro sensor before the posture of thehead is acquired. This minimizes any error in detecting the angle of theposture so as to prevent the problem of losing the sense of equilibriumas much as possible.

FIG. 5 depicts a configuration of functional blocks in the informationprocessing apparatus 10 and the HMD 18. The functional blocks depictedin FIG. 5 are implemented in a hardware configuration that may includethe CPU, GPU, various memories, display device, speakers, light-emittingelements, and sensors as illustrated in FIGS. 2 and 3. These functionalblocks are also implemented in software as programs that are loadedtypically from a recording medium into memory to provide a data inputfunction, a data holding function, an image processing function, and acommunication function, for example. Thus it will be understood by thoseskilled in the art that these functional blocks are realized by hardwarealone, by software alone, or by a combination of both in diverse formsand are not limited to any of such forms.

The information processing apparatus 10 includes an input informationacquiring section 72 that acquires information input from the inputdevice 14 and the HMD 18, a captured image acquiring section 74 thatacquires captured image data from the imaging device 12, a posture dataacquiring section 76 that acquires data representing the posture of theuser's head, an image analyzing section 78 that analyzes a capturedimage to track a target object, an information processing section 80that performs information processing in accordance with the applicationbeing executed such as a video game, and an output data transmittingsection 82 that transmits output data to the HMD 18 and the flat-screendisplay device 16.

The HMD 18 includes an output data receiving section 94 that receivesoutput data transmitted from the information processing apparatus 10, ameasuring section 90 composed of motion sensors and a communicationmechanism, a light-emitting section 92 made up of light-emittingelements and a mechanism for controlling the emission of lighttherefrom, a display processing section 98 that displays imagesextracted from the output data received by the output data receivingsection 94, and an audio processing section 96 that outputs soundsextracted from the output data.

The input information acquiring section 72 in the information processingapparatus 10 acquires detailed user operations from the input device 14.The user operations may include selection of the application to beexecuted, starting and ending of processing, input of commands, andother operations commonly performed during information processing.

Depending on the content of the information acquired from the inputdevice 14, the input information acquiring section 72 feeds theinformation to the captured image acquiring section 74 or to theinformation processing section 80. Also, the input information acquiringsection 72 receives motion sensor output values from the measuringsection 90 in the HMD 18 and feeds the received values to the posturedata acquiring section 76.

The captured image acquiring section 74 acquires at a predeterminedframe rate the data of the captured image such as a stereoscopic imageobtained by the imaging device 12 through video imaging. Also, thecaptured image acquiring section 74 may perform control to start or endimaging by the imaging device 12 in accordance with a processing startor end request obtained by the input information acquiring section 72from the user. Depending on the result of the processing by theinformation processing section 80, the captured image acquiring section74 may further control the type of data to be acquired from the imagingdevice 12.

On the basis of the sensor output values from the measuring section 90in the HMD 18, the posture data acquiring section 76 acquires theposture and movements of the user's head at a predetermined rate andfeeds what is acquired to the information processing section 80. Theposture data acquiring section 76 includes a correction data storingsection 84 acting as a storage area to store the data necessary forcorrection upon acquisition of the posture from the sensor output values(the data will be called “correction data” hereunder), a posturecalculating section 86 that calculates the posture of the head from thesensor output values using the correction data, and a correction dataupdating section 88 that updates the correction data as needed. In thepresent embodiment, the correction is performed on two kinds of error:(1) an error stemming from the initial mounting of motion sensors; and(2) an error gradually taking place due to deterioration with age of themotion sensors. Specific ways to perform the correction will bediscussed later.

The data for correcting the error in the category of (1) above isacquired during manufacturing of the HMD 18 and is stored into thecorrection data storing section 84 typically at the time of shipment.The data for correcting the error in the category of (2) above isprovided by the correction data updating section 88 updating, as needed,the correction data stored in the correction data storing section 84from the beginning. At this point, the correction data updating section88 determines whether it is necessary to update the correction data bycomparing two postures: the posture of the HMD 18 acquired as describedabove by the image analyzing section 78 from multiple marker images inthe captured image, and the posture calculated by the posturecalculating section 86. If the difference between the two postures issignificantly large, the correction data updating section 88 updates thecorrection data in such a manner that the same posture as that acquiredthrough image analysis will also be obtained from the motion sensoroutput values.

In subsequent corrections, the posture calculating section 86 uses theupdated correction data. If there is a possibility that multiple HMDs 18may be connected with the information processing apparatus 10, thecorrection data storing section 84 may store the correction data inassociation with individual identification information related to eachof the HMDs 18 so that the correction data will be managed with respectto each HMD. If the initial values of the correction data are stored inan internal memory of the HMD 18 for example, the posture data acquiringsection 76 loads the initial values retrieved from the memory into thecorrection data storing section 84 via the input information acquiringsection 72 typically at the time of the initial connection.

The image analyzing section 78 detects the images of the markers on theHMD 18 from each frame of the captured image and acquires therefrom theposition and posture of the user's head in the real space. The positionof the head in the real space is identified by the distances from theimaging plane to the markers calculated in accordance with thepositional relations between the relevant marker images in thestereoscopic image and by the positions of the marker images on theimage plane. Existing techniques may be used in the process ofidentifying the position of the target object and in the process oftracking the object. The image analyzing section 78 feeds theinformation related to the acquired position and posture consecutivelyto the information processing section 80.

At a predetermined timing or at a point in time when requested by thecorrection data updating section 88 in the posture data acquiringsection 76, the image analyzing section 78 supplies the correction dataupdating section 88 with the information related to the posture of theHMD 18 acquired on the basis of the number of marker images and thepositional relations therebetween in the captured image. In turn, thecorrection data updating section 88 updates as needed the correctiondata stored in the correction data storing section 84.

The information processing section 80 performs predetermined informationprocessing by suitably integrating the data about the head postureacquired by the posture data acquiring section 76 and the data about theposition and posture of the user obtained by the image analyzing section78. Most typically, a virtual world may be presented by advancing theongoing game and varying its visual field in response to the movementsof the user's head as described above. As long as the posture of thehead is used, the content of information processing is not limited toanything specific. The information processing section 80 outputs theresults of the information processing thus carried out as output datarepresenting images and sounds. The output data transmitting section 82successively acquires the output data generated by the informationprocessing section 80 and, after modifying the acquired data as needed,feeds the data to at least either the HMD 18 or the flat-screen displaydevice 16.

The output data receiving section 94 in the HMD 18 receives the outputdata from the information processing apparatus 10. Given the outputdata, the output data receiving section 94 feeds video data and audiodata extracted therefrom to the display processing section 98 and to theaudio processing section 96, respectively. As a result, the displaypanel included in the display processing section 98 outputs images, andthe speakers included in the audio processing section 96 output sounds.The display processing section 98 may cut out a portion of the imagethus generated for example and transmit the data of the cut-out portionto the flat-screen display device 16.

The measuring section 90, which includes the motion sensors 64 and thecommunication section 62 depicted in FIG. 3, transmits output values ofthe motion sensors 64 to the information processing apparatus 10 at apredetermined rate. The output values include such information as therates of acceleration on a predetermined number of axes measured by theacceleration sensor and an angular velocity measured by the gyro sensor.The light-emitting section 92, which includes the light-emitting section66 depicted in FIG. 3, causes it to function as light-emitting markersemitting light in a predetermined color. The color of emitted light maybe designated by the information processing apparatus 10. In this case,the output data receiving section 94 acquires from the informationprocessing apparatus 10 the data for designating the emitted-light colorand sends the acquired data to the light-emitting section 92. Forexample, the emitted-light color may be changed in accordance with theuser's identification information. This makes it possible to distinguishthe heads of multiple users by emitted-light color. Likewise, theinformation processing apparatus 10 may further designate theemitted-light color of markers attached to the input device 14.

FIG. 6 is a schematic diagram explaining typical relations betweendisplay images on the one hand and rates of acceleration targeted formeasurement with the acceleration sensor on the other hand. This exampledepicts a user 150 wearing on the head the HMD 18 equipped with athree-axis acceleration sensor. The example indicates how the visualfield of a virtual world 152 is changed in accordance with the postureof the user's head. Suppose that in a reference orthogonal coordinatesystem with a gravity vector g oriented vertically downward (where an X₀axis and a Y₀ axis define a horizontal plane and a Z₀ axis is in thevertical direction), the X, Y, and Z axes of the acceleration sensor areinclined by angles θ, ψ, and ϕ, respectively. On that assumption, therelations between the rates of acceleration A(X), A(Y), and A(Z) to bemeasured on the one hand and the angles of the axes on the other handare defined as follows:

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{{\theta = {\tan^{- 1}\left( \frac{A(X)}{\sqrt{{A(Y)}^{2} + {A(Z)}^{2}}} \right)}}{\psi = {\tan^{- 1}\left( \frac{A(Y)}{\sqrt{{A(X)}^{2} + {A(Z)}^{2}}} \right)}}{\phi = {\tan^{- 1}\left( \frac{\sqrt{{A(X)}^{2} + {A(Y)}^{2}}}{A(Z)} \right)}}} & \left( {{Expression}\mspace{14mu} 1} \right)\end{matrix}$

The posture of the acceleration sensor is obtained using the rates ofacceleration on the axes measured as defined by the expression 1 above.The direction of the user's face is obtained by the gyro sensor. In thismanner, the posture of the head of the user 150 is determined, and so isa directional vector v of the direction the user faces. A screencoordinate system is then determined accordingly, with the virtual word152 projected onto a screen 154 set up corresponding to the system. Thispermits display of a virtual-world image of which the visual field ischanged in keeping with the direction the user faces.

It is to be noted that the output values from the acceleration sensorcontain an error stemming from individual differences between sensorsand from the installed accuracy of substrates. Generally, this requiresperforming initial calibration upon manufacture of the accelerationsensor to thereby determine correction formulas for obtaining effectiverates of acceleration from the output values. If the output values fromthe acceleration sensor on the axes are assumed to be A_(out)(X),A_(out)(Y), and A_(out)(Z), then the correction formulas are given asfollows:A(X)=S _(X) ×A _(out)(X)+B _(X)A(Y)=S _(Y) ×A _(out)(Y)+B _(Y)A(Z)=S _(Z) ×A _(out)(Z)+B _(Z)where, S_(X), S_(Y), and S_(Z) stand for scale values and B_(X), B_(Y),and B_(Z) for bias values. These values constitute part of thecorrection data.

FIG. 7 is a schematic diagram explaining the technique for obtainingcorrection formulas through calibration. The calibration involvesacquiring the output values of the acceleration sensor fixed with jigsin six directions as illustrated. Specifically, in the states (a) and(b) of FIG. 7 in which the positive direction of the X axis isvertically upward and downward individually, the output valuesA_(out)(X+) and A_(out)(X−) on the X axis are acquired. In the states(c) and (d) in which the positive direction of the Y axis are verticallyupward and downward individually, the output values A_(out)(Y+) andA_(out)(Y−) on the Y axis are measured. In the states (e) and (f) inwhich the positive direction of the Z axis is vertically upward anddownward individually, the output values A_(out)(Z+) and A_(out)(Z−) onthe Z axis are measured.

For example, the rate of acceleration A(X) on the X axis is expected tobe −g in the state (a) and +g in the state (b), so that the followingexpressions are formulated with respect to these states:−g=S _(X) ×A _(out)(X+)+B _(X)+g=S _(X) ×A _(out)(X−)+B _(X)The scale value S_(X) and the bias value B_(X) are obtained by solvingthe above simultaneous equations. The same applies to the Y axis and theZ axis as well. As a result of this, the scale values S_(X), S_(Y), andS_(Z) and the bias values B_(X), B_(Y), and B_(Z) are obtained.

Theoretically, the calibration carried out as described above improvesthe accuracy in acquiring the posture. However, when the posture isreflected in the display image as with the present embodiment, even aminuscule error of approximately 1 degree can give the user theabove-mentioned feeling of discomfort. In particular, a display setupsuch as the HMD offering a sense of immersion can adversely affect theuser's physical condition. Even after the above-described initialcalibration, there still can occur a small error stemming from diversefactors including the accuracy in using jigs or making measurementsduring calibration and the accuracy in attaching the sensor to the HMD18.

Accordingly, more correction data is acquired with the accelerationsensor attached to the HMD 18 so that the error may be minimized.Specifically, the mounting angle of the acceleration sensor is assumedto include a minuscule error. On that assumption, rotation correction iscarried out on the posture obtained from the acceleration in such amanner as to offset the error. Thus on the basis of the posture of theacceleration sensor, the posture of not only the HMD 18 but also theuser's head is acquired accurately. The error stemming from theabove-mentioned factors following sensor installation also applies tothe other sensors such as the gyro sensor included in the motion sensors64. The same correction data may then be used to correct the outputvalues from the other sensors as well. This permits more accurateacquisition of the directional vector v of the direction the user faces,for example. FIG. 8 is a schematic diagram explaining the principle ofperforming rotation correction. First, with the acceleration sensormounted on the HMD 18, the HMD 18 is fixed corresponding to each of thedirections (a) to (f) in FIG. 7. Then the output values on the threeaxes of the acceleration sensor are acquired.

These output values are corrected using the scale values S_(X), S_(Y),and S_(Z) and the bias values B_(X), B_(Y), and B_(Z) mentioned above.If the acceleration sensor is installed with no error, it is in the samestate as that of the initial calibration described above. In that case,the rates of acceleration A(X), A(Y), and A(Z) after the correction areeither +g or −g in the X axis direction in the states (a) and (b) ofFIG. 7, in the Y axis direction in the states (c) and (d), and in the Zaxis direction in the states (e) and (f). In the other axial directions,the rates of acceleration A(X), A(Y), and A(Z) are 0. On the other hand,where there exits an error, even if the HMD 18 is fixed corresponding tothe states (a) to (f) in FIG. 7, the axes have angles relative to thevertical direction due to a slight inclination of the accelerationsensor. As a result, the rates of acceleration turn out to be some othervalues.

That is, given these rates of acceleration, the axial inclinations ofthe acceleration sensor are obtained as acceleration vectors in the sixstates in which the initial coordinate axes of the HMD 18 are made tocoincide with the gravity vector. The inclination angles of the axes arerelated to the rates of acceleration A(X), A(Y), and A(Z) in a mannerdefined by the expression 1 above. It is assumed here that with the HMD18 fixed corresponding to the states (a) to (f) in FIG. 7, theacceleration vectors given by the acceleration sensor are V(X+), V(X−),V(Y+), V(Y−), V(Z+), and V(Z−). In FIG. 8, with these states integrated,the inclinations of the acceleration vectors are depicted simultaneouslyrelative to an X_(H) axis, a Y_(H) axis, and a Z_(H) axis of the HMD 18.Because there is a difference θ of 180 degrees between the states (a)and (b) in FIG. 7, the acceleration vectors V(X+) and V(X−) aretheoretically in opposite directions but on the same straight line. Thisline constitutes the X axis of the acceleration sensor.

Likewise, the acceleration vectors V(Y+) and V(Y−) constitute the Y axisof the acceleration sensor, and the acceleration vectors V(Z+) and V(Z−)make up its Z axis. If there is no error in mounting the accelerationsensor, its X, Y, and Z axes coincide with the coordinate axes of theHMD 18. If the acceleration sensor is slightly inclined for example, theorthogonal coordinates on the X, Y, and Z axes occur rotated by theamount of the inclination with respect to the orthogonal coordinates ofthe HMD 18. To counter this aberration, error angles σ_(θ), σ_(ψ), andσ_(ϕ) of the axes are obtained beforehand and stored into the correctiondata storing section 84.

In operation, the posture calculating section 86 using the expression 1above obtains the posture from the rates of acceleration based on theoutput values from the acceleration sensor, and subtracts the errorsinvolved from the respective angles to accurately calculate the postureof not only the HMD 18 but also the user's head in particular. Similarcorrection is carried out on the angle information presented by anyother sensor. It might not be possible, however, to obtain the linearityof the acceleration vectors constituting any one axis of theacceleration sensor or the orthogonality between its axes due to factorsother than the installation error. For example, the acceleration vectorsV(Z+) and V(Z−) may not constitute a straight line, or either of the twovectors may not be clearly orthogonal to the X axis or the Y axis.

In that case, the acceleration vectors, if rotated uncorrected, fail tocoincide with the orthogonal coordinate axes of the HMD 18. Thisaberration may be countered by removing these acceleration vectors fromthe calculation of the error angles σ_(θ), σ_(ψ), and σ_(ϕ). Forexample, any one axis considered aberrant may be identified in terms oflinearity and orthogonality. The remaining two axes may then be rotatedto obtain their error angles in such a manner that the two axes coincidewith the corresponding axes of the HMD 18. Depending on the content ofinformation processing such as video games, the axis subject to a largeangular variation may be limited. In this case, the error angle of thelimited axis may be obtained in such a manner that the axis ispreferentially made to coincide with the corresponding axis of the HMD18. Such measures effectively improve the accuracy of posture detection.

Alternatively, with the angle of each acceleration vector acquiredrelative to the corresponding axis of the HMD 18, the error angles ofthe axes involved may be averaged to find a mean error angle for eachaxis. As another alternative, the error angle may be calculated from theamount of rotation on a given axis coming closest to any axis of the HMD18. It is also possible to exclude, from the posture calculation duringoperation, any axis on which the rate of acceleration is clearlyaberrant in terms of linearity of the acceleration vector to be had onthe same axis or in terms of orthogonality between the axes involved.

Explained below is the operation carried out by the configurationdescribed above. FIG. 9 is a flowchart depicting the procedure of aposture acquiring process performed by the posture data acquiringsection 76, the posture acquiring process including a correction dataupdating process executed by the correction data updating section 88.Incidentally, the posture data acquiring section 76 may generate orintegrate information as needed from the measurements made by the othersensors, not depicted, besides the acceleration sensor, the sensorsbeing included in the motion sensors mounted on the HMD 18.

First, the posture calculating section 86 acquires the output valuesfrom the acceleration sensor of the HMD 18 (S10). The posturecalculating section 86 corrects the output values by referencing thecorrection data stored in the correction data storing section 84 (S12).Specifically, the posture calculating section 86 calculates theeffective rates of acceleration by substituting the output values on theaxes for correction formulas that include scale values and bias valuesprepared in advance. At this point, where multiple HMDs 18 are allowedto be connected with the information processing apparatus 10, theposture calculating section 86 first acquires the individualidentification information related to each HMD 18 before referencing thecorrection data associated with the acquired individual identificationinformation. This applies not only to the scale values and bias valuesbut also to the angle errors.

The posture calculating section 86 then calculates the inclinationangles θ, ψ, and ϕ relative to the reference orthogonal coordinatesystem, i.e., calculates the posture using the expression 1 above (S14).At this point, the posture calculating section 86 obtains the headposture accurately from the posture of the acceleration sensor bycarrying out rotation correction on the output values by the amounts ofthe error angles σ_(θ), σ_(ψ), and σ_(ϕ) stored in the correction datastoring section 84. Similar rotation correction is carried out on theangle-related output values from the other sensors. The postureinformation thus calculated is output to the information processingsection 80 (S16). Meanwhile, the correction data updating section 88acquires from the image analyzing section 78 the information related tothe posture of the HMD 18 obtained as a result of analyzing the capturedimage (S18). This information is acquired, as mentioned above, typicallyby geometric calculations using computer graphics technology based onthe number of marker images of the HMD 18 and on the positionalrelations between the marker images in the captured image.

Next, the correction data updating section 88 compares the postureobtained in step S14 based on the motion sensors with the postureacquired as a result of image analysis to determine whether or not thedifference therebetween is larger than a predetermined threshold value(S20). For example, angles are compared on each of the axes. If there isany axis on which the difference between the compared angles is largerthan the predetermined threshold value, that axis is identified. Ifthere is no such axis, the processing is terminated (N in S20). If thereis an axis on which the difference between the compared angles is largerthan the predetermined threshold value (Y in S20), the correction dataupdating section 88 updates the correction data in the correction datastoring section 84 in such a manner that the angle conforming to theposture obtained by image analysis is acquired (S22). For example, thecorrection data updating section 88 may back-calculate the expression 1above to obtain rates of acceleration that permit acquisition of theangle based on image analysis, and update the bias values in such amanner as to obtain these rates of acceleration from the output valuesof the acceleration sensor. However, the target to be corrected is notlimited to the bias values. The scale values and the error angles foruse in rotation correction may also be considered the target to becorrected.

Repeating steps S10 to S22 at a predetermined frequency permitscontinuous acquisition of the posture of not only the HMD 18 but alsothe user's head. The acquired posture is reflected in the result ofinformation processing such as in the display image. Even if thecorrection data is subject to secular changes even during operation, theeffects of such changes are minimized. Although the illustrated exampledepicts that the process of calculating and outputting the posture andthe process of updating correction data are performed in series, the twoprocesses may alternatively be carried at independent timings inpractice. For example, the posture information based on image analysismay be accumulated for a predetermined time period. When the differencebetween the posture based on the accumulated information and the posturederived from the acceleration sensor has come to be larger than apredetermined threshold value for a predetermined time period, thecorrection data may be updated.

As described above, where the posture information is corrected on thebasis of the result of image analysis, the correction data forpermitting acquisition of the corrected posture is obtained and stored.The data is used next time the processing is carried out so that theaccuracy of posture acquisition will be sustained. Because thedifference between the stored posture information and the postureinformation based on image analysis is corrected in a short time, theamount of correction at one time is reduced even if secular changescontinue. There is thus no need to carry out time-consuming, massivecorrection. Such secular changes may appear only on a particular axis inconsequence of the process of manufacturing acceleration sensors, forexample.

If information identifying an axis that is subject to larger secularchanges than the other axes or which proves aberrant when measurementsare made thereon is obtained typically from product informationdescribing the characteristics of motion sensors, or if such an axis isdetected by actual measurements, the motion sensors may be mounted onthe HMD 18 in such a manner that the aberrant axis is oriented in aparticular direction in consideration of easy human perception. Forexample, the aberrant axis is oriented vertically so that its secularchanges will be minimized in detecting angles relative to a horizontalplane. This stabilizes the angle in the horizontal direction that ismost sensitively perceived, which implements the display of images freeof discomfort for a long time.

If there is an order in which the amounts of secular changes or theaccuracy levels of the three axes are presented, the acceleration sensormay be mounted so that its axes are set up in descending order ofamounts of secular changes or in ascending order of accuracy levels,e.g., in a vertical, front-back, or crosswise direction. When the axesare thus established in descending order of difficulty in perceiving theaberration thereof, the adverse effects of the characteristics of theacceleration sensor are minimized. The mounting direction of theacceleration sensor may be determined in consideration of least adverseeffects of the aberrant axis, with the inclination in a particulardirection not reflected in the result of the information processing suchas in a video game, or with the user's movements on the aberrant axiskept minimal for example, in addition to taking into account thedifficulty of perceiving the aberration.

With the above-described embodiment implementing techniques fordetecting the posture of the target object using motion sensors,rotation errors are acquired and stored besides the scale values andbias values for converting the output values of the acceleration sensorinto rates of acceleration. In operation, the posture of the targetobject is accurately acquired using the stored values so as to carry outrotation correction on the angle information obtained from theacceleration sensor and other sensors. This makes it possible tominimize the effects of the errors stemming from various factors such asindividual differences of the sensors, installed accuracy levels, andaccuracy of jigs for calibration.

In the setup where the motion sensors are attached to the HMD so thatthe HMD displays images of which the visual field is varied in responseto the user's head movements, the user is liable to experience a feelingof discomfort as well as worsening of physical conditions if presentedwith a world depicting even a minuscule error of approximately 1 degreewith respect to reality. This aberration is effectively countered by theabove-described configuration. In operation, the correction data isupdated as needed in comparison with the posture information obtained bymeans other than the motion sensors about the same target object. Forexample, with the motion sensors attached to the HMD, multiple markersare also mounted on the HMD. The posture of the user's head is thenacquired separately on the basis of the number of marker images and thepositional relations therebetween in the captured image.

Where the separately acquired posture information is used to update thecorrection data as needed, the effects of secular changes in the sensorsare minimized, and an appropriate state is easily sustained. If it isknown beforehand that secular changes are more pronounced or that theaccuracy of measurement is lower on a certain axis than on the otheraxes, then the aberrant axis is set up vertically for example, or in aparticular direction selected to be least amenable to perception. Thisminimizes the adverse effects on the user such as a feeling ofdiscomfort with regard to images displayed in the visual fieldreflecting the posture. Because the relatively inexpensive accelerationsensor may be used, the manufacturing costs of the system are reduced.

The present invention has been described above in conjunction with aspecific embodiment. It is to be understood by those skilled in the artthat suitable combinations of constituent elements and processes of theembodiment described above as an example may lead to further variationsof the present invention and that such variations also fall within thescope of the present invention.

For example, although the present embodiment has been described as beingimplemented through the combination of rotation correction based onangle errors, updating of correction data using the result of imageanalysis, and determination of the mounting direction reflecting thecharacteristics of sensor axes, such constituent operations may each beimplemented independently. In such a case, the independently implementedoperations still permit stable and highly precise acquisition of theposture information.

REFERENCE SIGNS LIST

8 Information processing system, 10 Information processing apparatus, 12Imaging device, 14 Input device, 16 Flat-screen display device, 18 HMD,22 CPU, 24 GPU, 32 Communication section, 54 Display section, 56 Audiooutput section, 62 Communication section, 64 Motion sensor, 66Light-emitting section, 72 Input information acquiring section, 74Captured image acquiring section, 76 Posture data acquiring section, 78Image analyzing section, 80 Information processing section, 82 Outputdata transmitting section, 84 Correction data storing section, 86Posture calculating section, 90 Measuring section, 92 Light-emittingsection, 94 Output data receiving section, 96 Audio processing section,98 Display processing section.

INDUSTRIAL APPLICABILITY

As described above, the present invention is applicable to a gamemachine, an information processing apparatus, an object recognitionapparatus, a head-mounted display, and a system that includes any one ofthese devices and apparatuses, among others.

The invention claimed is:
 1. An information processing apparatuscomprising: an input information acquiring section configured to acquireoutput values from an acceleration sensor mounted on a target object; aposture data acquiring section configured to acquire posture informationrelated to the target object by calculating rates of acceleration onaxes of the acceleration sensor based on the output values so as toobtain inclination angles of the axes based on the calculated rates ofacceleration; and an information processing section configured toperform information processing using the posture information beforeoutputting output data generated as a result of the informationprocessing, wherein, on the basis of angle errors between the axes ofthe acceleration sensor on the one hand and reference axes given to thetarget object on the other hand, the posture data acquiring sectioncorrects the posture information related to the acceleration sensor inaccordance with the posture information related to the target object. 2.The information processing apparatus according to claim 1, furthercomprising: a correction data storing section configured to store, asthe angle errors, amounts of angular change of multiple axes of theacceleration sensor being rotated simultaneously until the axes coincidewith the corresponding axes from among multiple reference axes given tothe target object, wherein the posture data acquiring section performsthe correction by referencing the angle errors stored in the correctiondata storing section.
 3. The information processing apparatus accordingto claim 2, wherein the correction data storing section stores the angleerrors in association with individual identification information relatedto a device as the target object mounted with the acceleration sensor,and the posture data acquiring section acquires the individualidentification information related to the device mounted with theacceleration sensor from which the output values were acquired, theposture data acquiring section further referencing the angular errorsassociated with the acquired individual identification information. 4.The information processing apparatus according to claim 2, furthercomprising: a correction data updating section configured to acquireposture information generated by means other than the accelerationsensor regarding the same target object and compare the postureinformation thus acquired with the posture information acquired by theposture data acquiring section in order to update correction data foruse until the posture data acquiring section acquires the postureinformation from the output values, the correction data updating sectionfurther storing the updated data into the correction data storingsection.
 5. The information processing apparatus according to claim 4,wherein the correction data updating section updates at least one ofparameters for use in correction formulas for calculating the rates ofacceleration based on the output values.
 6. The information processingapparatus according to claim 1, wherein the posture data acquiringsection corrects the posture information using the angle errorscalculated beforehand by assuming as an axis of the acceleration sensora line formed by connecting two acceleration vectors presented by theacceleration sensor in a state where reference axes given to the targetobject are made to coincide with a gravity vector in positive andnegative directions.
 7. The information processing apparatus accordingto claim 1, wherein the posture data acquiring section corrects, on thebasis of the angle errors, the output values from another sensor mountedalong with the acceleration sensor with a view to measuringangle-related values, the posture data acquiring section furtherintegrating the corrected output values with the posture information. 8.An information processing system comprising: a head-mounted display; andan information processing apparatus configured to establish connectionwith the head-mounted display and to generate data of a display image,wherein the information processing apparatus includes an inputinformation acquiring section configured to acquire output values froman acceleration sensor mounted on the head-mounted display, a posturedata acquiring section configured to acquire posture information relatedto the head-mounted display by calculating rates of acceleration on axesof the acceleration sensor based on the output values so as to obtaininclination angles of the axes based on the calculated rates ofacceleration, and an information processing section configured togenerate the data of the display image having a visual view beingchanged in accordance with posture using the posture information, theinformation processing section further outputting the display image datato the head-mounted display, and the posture data acquiring sectioncorrects the posture information related to the acceleration sensor inaccordance with the posture information related to the head-mounteddisplay on the basis of angle errors between the axes of theacceleration sensor on the one hand and reference axes given to thehead-mounted display on the other hand.
 9. The information processingsystem according to claim 8, wherein the acceleration sensor is mountedon the head-mounted display in a direction corresponding to thecharacteristics acquired of each of the axes.
 10. The informationprocessing system according to claim 9, wherein the acceleration sensoris mounted in such a manner that an axis of the acceleration sensorsubject to a larger amount of secular change than the other axes is in aperpendicular direction of the head-mounted display worn by a user. 11.An information processing method comprising: acquiring output valuesfrom an acceleration sensor mounted on a target object; acquiringposture information related to the target object by calculating rates ofacceleration on axes of the acceleration sensor based on the outputvalues so as to obtain inclination angles of the axes based on thecalculated rates of acceleration; and performing information processingusing the posture information before outputting output data generated asa result of the information processing to another apparatus, wherein, onthe basis of angle errors between the axes of the acceleration sensor onthe one hand and reference axes given to the target object on the otherhand, the acquiring the posture information corrects the postureinformation related to the acceleration sensor in accordance with theposture information related to the target object.